ATFormer: Advanced transformer for medical image segmentation

Combining transformers and convolutional neural networks is considered one of the most important directions for tackling medical image segmentation problems. To learn the long-range dependencies and local contexts, previous approaches embedded a convolutional layer into feedforward neural network inside the transformer block. However, a common issue is the instability during training since large differences in amplitude across layers by pre-layer normalization. Furthermore, multi-scale features were directly fused using the transformer from the encoder to decoder, which could disrupt valuable information for segmentation. To address these concerns, we propose Advanced TransFormer (ATFormer), a novel hybrid architecture that combines convolutional neural networks and transformers for medical image segmentation. First, the traditional transformer block has been refined into an Advanced Transformer Block, which adopts post-layer normalization to obtain mild activation values and employs the scaled cosine attention with shifted window for accurate spatial information. Second, the Progressive Guided Fusion module is introduced to make multi-scale features more discriminative while reducing the computational complexity. Experimental results on the ACDC, COVID-19 CT-Seg, and Tumor datasets demonstrate the significant advantage of ATFormer over existing methods that rely solely on convolutional neural networks, transformers, or their combination.

A New Pneumonia Detection Model Based on Transformer with Improved Self-Attention Mechanism

Clinical diagnosis based on computed tomography (CT) could be used, as part of diagnosis standard of COVID-19 pneumonia. Addressing the problem that accuracy of CT-based traditional pneumonia classification diagnosis models is relatively low when employed for classification of community-acquired pneumonia (CP), COVID-19 pneumonia (NCP) and normal cases, a new network model is proposed which combines application of Swin Transformer and multi-head axial self-attention (MASA) mechanism, to analyze CT images and make intelligence-assisted diagnosis. The method in detail is to partially replace traditional multi-head self-attention (MSA) mechanism in encoders of Swin Transformer by MASA. The improved model is applied to train and test on commonly used pneumonia CT dataset CC-CCII. The results show that the proposed network outperforms traditional networks ResNet50 and Vision Transformer in indicators of accuracy, sensitivity and F1-measure. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

MPSKT: Multi-head ProbSparse Self-Attention for Knowledge Tracing

Over the past two years, COVID-19 has led to a widespread rise in online education, and knowledge tracing has been used on various educational platforms. However, most existing knowledge tracing models still suffer from long-term dependence. To address this problem, we propose a Multi-head ProbSparse Self-Attention for Knowledge Tracing(MPSKT). Firstly, the temporal convolutional network is used to encode the position information of the input sequence. Then, the Multi-head ProbSparse Self-Attention in the encoder and decoder blocks is used to capture the relationship between the input sequences, and the convolution and pooling layers in the encoder block are used to shorten the length of the input sequence, which greatly reduces the time complexity of the model and better solves the problem of long-term dependence of the model. Finally, experimental results on three public online education datasets demonstrate the effectiveness of our proposed model. © 2022 Association for Computing Machinery.

An Intelligent Athlete Signal Processing Methodology for Balance Control Ability Assessment with Multi-Headed Self-Attention Mechanism

In different kinds of sports, the balance control ability plays an important role for every athlete. Therefore, coaches and athletes need accurate and efficient assessments of the balance control ability to improve the athletes’ training performance scientifically. With the fast growth of sport technology and training devices, intelligent and automatic assessment methods have been in high demand in the past years. This paper proposes a deep-learning-based method for a balance control ability assessment involving an analysis of the time-series signals from the athletes. The proposed method directly processes the raw data and provides the assessment results, with an end-to-end structure. This straight-forward structure facilitates its practical application. A deep learning model is employed to explore the target features with a multi-headed self-attention mechanism, which is a new approach to sports assessments. In the experiments, the real athletes’ balance control ability assessment data are utilized for the validation of the proposed method. Through comparisons with different existing methods, the accuracy rate of the proposed method is shown to be more than 95% for all four tasks, which is higher than the other compared methods for tasks containing more than one athlete of each level. The results show that the proposed method works effectively and efficiently in real scenarios for athlete balance control ability evaluations. However, reducing the proposed method’s calculation costs is an important task for future studies.

Self-Attention Mechanism of RoBERTa to Improve QAS for e-health Education

This paper aims to discuss about the establishment of a health education system in the form of a Question Answer System (QAS) related to the current COVID-19 pandemic. QAS allows users to state information needs in the form of natural language questions, and then this system will return short text quotes or sentence phrases as answers. This is due to the tendency for recipients of information to more easily understand news/information when they can answer questions that may arise in their minds. The approach used was self-attention mechanism such as a Robustly Optimized BERT Pretraining Approach (RoBERTa), a method for question answering with span-based training that predicting the starting limit for answer start and the end limit for the answer index. The final results using 835 non-description questions, the best evaluation value on the training data showed the exact match of 91.7% and F1 value of 93.3%. RoBERTa tends to show the better results on non- description questions or questions with short answers compared to the description questions with complex answers. © 2021 IEEE.

LRSAR-Net Semantic Segmentation Model for Computer Aided Diagnosis for Covid-19 CT Image

Since the outbreak of the Covid-19 epidemic in the world in late 2019, all countries in the world are under the threat of epidemic. Covid-19 invades the body's respiratory system, causing lung infection or even death. Computed Tomography (CT) is a routine method for doctors to diagnose patients with pneumonia. In order to improve the efficiency of doctors in diagnosing patients with new crown infection, this paper proposes a semantic segmentation network LRSAR-Net based on low rank tensor self-attention reconstruction, in which the low rank tensor self-attention reconstruction module is used to obtain long-range information. The low rank tensor self-attention reconstruction module mainly includes three parts: low rank tensor generation sub module, low rank self-attention sub module and high rank tensor reconstruction module. The low rank tensor self-attention module is divided into multiple low rank tensors, the low rank self-attention feature map is constructed, and then the multiple low rank tensor attention feature maps are reconstructed into a high rank attention feature map. The self-attention module obtains long-range semantic information by calculating the similarity matrix. Compared with the traditional self-attention module Non Local, the low rank tensor self-attention reconstruction module has lower computational complexity and faster computing speed. Finally, this paper compares with other excellent semantic segmentation models to reflect the effectiveness of the model. © 2022, Science Press. All right reserved.

Densely connected attention network for diagnosing COVID-19 based on chest CT.

BACKGROUND: To fully enhance the feature extraction capabilities of deep learning models, so as to accurately diagnose coronavirus disease 2019 (COVID-19) based on chest CT images, a densely connected attention network (DenseANet) was constructed by utilizing the self-attention mechanism in deep learning. METHODS: During the construction of the DenseANet, we not only densely connected attention features within and between the feature extraction blocks with the same scale, but also densely connected attention features with different scales at the end of the deep model, thereby further enhancing the high-order features. In this way, as the depth of the deep model increases, the spatial attention features generated by different layers can be densely connected and gradually transferred to deeper layers. The DenseANet takes CT images of the lung fields segmented by an improved U-Net as inputs and outputs the probability of the patients suffering from COVID-19. RESULTS: Compared with exiting attention networks, DenseANet can maximize the utilization of self-attention features at different depths in the model. A five-fold cross-validation experiment was performed on a dataset containing 2993 CT scans of 2121 patients, and experiments showed that the DenseANet can effectively locate the lung lesions of patients infected with SARS-CoV-2, and distinguish COVID-19, common pneumonia and normal controls with an average of 96.06% Acc and 0.989 AUC. CONCLUSIONS: The DenseANet we proposed can generate strong attention features and achieve the best diagnosis results. In addition, the proposed method of densely connecting attention features can be easily extended to other advanced deep learning methods to improve their performance in related tasks.